Gain from Strain? Assessing the Impact of User Fatigue on the Quality of Subjective MOS Ratings

This paper addresses a central question in subjective test design: ”What is the appropriate duration for QoE lab experiments that involve human participants?” Since user tests are time-consuming and costly, this is an important question as it exposes the fundamental challenge of maximizing test duration while avoiding problems arising from undue strain on test participants. We provide an extensive analysis of the influence that workload and fatigue have on the rating behavior of QoE test participants, based on empirical data. Our analysis is grounded on measurements and experiences made during two typical QoE lab studies which assessed the impact of different network conditions on perceived quality of data services (web browsing, file download). During both studies, we measured participants’ workload and fatigue in two complementary ways, subjectively by means of a questionnaire and objectively by capturing eye blink rate (EBR). Our main finding is that even after 90 minutes of active testing, the mean and standard deviation of participants’ MOS grades were not significantly affected by their temporal position within the test sessions (beginning vs. end). Thus, for comparable QoE lab user experiments, this duration represents a safe recommendation for balancing results quantity with results quality.